The present invention, in some embodiments thereof, relates to service oriented distributed data management with associated architecture and, more particularly, but not exclusively, to such a system that uses content and metadata to manage the data. Today the amount of storage keeps on growing exponentially, with major growth in new storage fronts such as cloud and BigData, and future growth trends with the emergence of the internet of things (IoT). In the new age data is dynamic, can move or replicate between different physical locations, can move between faster solid state storage or slower mechanical hard drives (HDD) based on usage patterns, and needs to be shared by many clients or applications, whether they are sensors, mobile phones, tablets, PCs, or application servers.
To date, most storage of data is block storage. Data is blindly arranged in blocks with no reference to the content. Multiple level storage can follow usage patterns as mentioned above to place more frequently used pieces of data in faster storage while keeping all of the data in main storage. Firewalls and other network control devices can inspect packets and direct data transfer according to the results, which may occasionally relate to the content, but there is no equivalent in data storage and management once beyond the bounds of networking.
The present trends suggest a different approach to storage, no longer silos of single vendor block storage mapped to a single client or application, but rather an adoption of highly scalable file or object based globally networked storage, which can serve many clients or applications, and connect to multiple local, remote, or cloud based storage resource islands/pools from a variety of vendors and be managed as a service.
Today there are some file sharing solutions in the form of NAS (Network Attached Storage) or even clustered/scale-out NAS solutions which have internal storage and support file access protocols, those however are limited in scale, provide only basic file management data services, and focus on local deployments.
Some new object storage solutions exist as local clusters or in the form of cloud storage, and examples include Amazon S3, Google Drive, and Dropbox, all of which are quite scalable, but are still limited in performance and in the data services that they provide, have limited consistency, and do not allow mixing various storage resources. The main use of those services is for backup and archiving and not as a form of enterprise storage or a way to unify different storage resources or protocols under the same governance.
Given the huge variety of potential clients and applications, each requiring different storage services, or different priorities, and the enormous scale and variety of the underlying storage resources, the management paradigm may be expected to change from the present day black box type of management which is agnostic to the data content.